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Abstract 

The scale and scope of online education increasingly expand. In tandem, interest grows 
among educators and scholars in understanding the personal and contextual factors that 
moderate the efficient design and effective delivery of an online course. This paper looks at a 
theoretically robust method, peer assessment administered by the Canvas learning 
management system, and evaluates its potential and performance in university-level hybrid 
and online courses. Cross-sectional data profile students’ views, both prior to as well as 
post-experience, of aspects of the peer review process. Frequency analyses, mean 
comparisons, and t-tests highlight students’ initial strong, positive agreement with the 
potential but then, post-experience, weaker support of the value of peer assessment. 
Controlling for time and place, in terms of hybrid versus online sections, did not moderate 
this outcome. Collectively, these results call for refining our interpretation of the utility of 
peer assessment to promoting student engagement. We evaluate the moderating influence of 
anonymity and the social dynamic of self-directed learning, particularly given the 
implications of the “anytime, anyplace” instrumentality of online education. We close with a 
series of pedagogical recommendations. 
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1. Introduction 

Research on the design and delivery of online education has focused on a panorama of 
dimensions, such as learning effectiveness, testing effects, academic honesty, test anxiety, 
social dynamics, student achievement, and learning strategies. Increasingly, complementary 
trends in the expanding technological functionality of learning management systems (LMS) 
along with growing interest in improving the integrity of the performance of online classes, 
compel evaluating pedagogies once regarded as impractical but now increasingly 
straightforward. 

For example, instructors look to the robust potential of peer assessments to set, support, and 
sustain a course context that promotes student engagement and learning effectiveness. In 
principle, by creating systematic opportunities for classmates to assess and constructively 
criticize each other’s works, the peer review process reportedly promotes engagement on a 
variety of intellectual, social, and operational fronts. Completing the assigned responsibility 
of formally assessing their classmates’ work, an often unprecedented charge, promotes 
several drivers of student engagement, including motivation, reflection, autonomy, 
self-efficacy, diplomacy, problem-solving, and responsibility (Boud & Falchikov, 2007; 
Stepanyan, Mather, Jones, & Lusuardi, 2009). Done well, Mulder, Pearce, and Baik (2014) 
conclude, peer reviews “nurture an impressively wide array of generic skills related to giving 
and accepting criticism, which are of fundamental importance in any professional 
workplace.” 

In principle, elucidations of the theoretical benefits of peer assessment are compelling. 
However, scant research has looked at how students perceive the opportunity of and then 
interpret the value of their direct experience with peer assessment over the course of a class 
(Mulder et ah, 2014; Nicol, Thomson, & Breslin, 2014). Once an understandable gap, current 
circumstances make it increasingly problematic. Traditionally, administering the peer 
assessment process within the context of a course, either on-campus or online, without the aid 
of custom-designed software imposed onerous logistics. Consequently, despite compelling 
educational, professional, and pedagogical merits, the typical student had few, if any, formal 
opportunities to review their classmates’ work (Lunstrom & Baker, 2009; Mulder et ah, 2014). 
Now, the progressive sophistication of cloud-based LMSs, by revolutionizing the practicality 
of real-time peer assessments, support wide-scale use. Indeed, LMSs’ improving functionality 
make it increasingly easier for instructors to administer and students to engage peer 
assessment procedures. 

Commensurately, improving our understanding of the potential for peer assessment 
procedures to enhance students’ engagement promises to bolster the effectiveness of online 
education. This supposition, of course, presumes that students expect and then experience 
meaningful utility from participating in the peer review process. Presently, however, 
empirical research on peer assessment falls short of unequivocally documenting whether 
students assigned peer review responsibilities anticipate it to be and then conclude it was a 
worthwhile, productive pedagogy. This paper aims to help fill this gap. 
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2. Setting a Design and Delivery Standard 

Translating the potential of peer reviews into purposeful, productive performance ultimately 
requires the effective design and efficient delivery of a useful activity (Boud, 2000; Mulder et 
ah, 2014). Expectedly, given the intrinsic ambiguity of the educational process, there is 
tremendous variance in setting a valid standard to evaluate the usefulness of potential course 
design and delivery options. Our interest in the performance of the peer review pedagogy in 
online and hybrid formats made us particularly mindful of the moderators of student learning 
in online environments. More precisely, after designing and delivering a series of hybrid and 
online courses, like others, the authors steadily but surely saw the performance implications 
and social dynamics of how students opted to engage activities, assignments, and assessments 
(Richardson & Newby, 2010). Always important to student performance, online education 
dramatically increases the importance of student engagement (Dixson, 2012; Carr, 2014). 
Hence, we heed research on student engagement and its assertion that it occurs when 
“students make a psychological investment in learning. They tty hard to learn what school 
offers. They take pride not simply in earning the formal indicators of success (grades), but in 
understanding the material and incorporating or internalizing it in their lives" (Newman, 
1992). Assessing student engagement, both as a construct and an outcome, allows us to go 
beyond anecdotes and course grades to gain knowledge about how students go about the 
process and procedures of learning. 

Pressing in any context, we believe engineering student engagement is particularly important 
in online formats. Research, anchored in traditional, face-to-face, classroom settings, 
identifies straightforward standards to spur students to engage peer review assignments. 
Promoting effective learning cycles, whereby students meaningfully engage the class as well 
as their classmates, calls for catalyzing the social connections that promote interpersonal 
collaboration (Boud & Falchikov, 2007). Working interactively enables peers to experience 
the in situ behavioral gestalt that supports the constructive relationships that influence the 
effectiveness of peer assessment (Greguras, Robie, & Bom, 2001). Furthennore, productive 
interactions with peers boost students’ diplomatic skills and negotiation competencies. This, 
in turn, contributes to the collaborative and participatory learning environment that supports 
effective review processes (Mulder et ah, 2014). Empirically, research documents the 
convergence between the potential and performance of peer review in the bricks-and-mortar 
classroom (Cartney, 2010; Mulder et ah, 2014). 

Presently, we cannot say the same with respect to the potential and performance of peer 
assessment in online environments. The unique milieu of an online class, particularly when 
framed in terms of orthodox social cognitive or self-determination perspectives, creates 
situational uncertainties regarding a range of personal and contextual variable (Pekrun, 2006; 
Artino, 2009). For example, Crews and Butterfield (2014) found that face-to-face interaction 
significantly affects personal and environmental contextual relationships. If so, for example, 
then how does the absence of conventionally defined face-time in an online environment 
affect students’ interpretation of the potential and performance of peer assessment 
opportunities? Consequently, educational constructs, extensively mapped in the bricks and 
mortar classroom, such as time, place, motivation, and cognitive engagement, presently defy 


3 


www.macrothink.org/jei 




Macrothink 

Institute™ 


Journal of Educational Issues 

ISSN 2377-2263 
2015, Vol. 1, No. 1 


straightforward specification in an online setting. Put directly, empirical research on peer 
assessment falls short of unequivocally documenting whether students who are assigned peer 
review responsibilities expect it to be and then conclude it was a worthwhile, productive 
pedagogy. And, absent objective information, we struggle to determine whether peer 
assessment procedures promote or, quite possibly, discourage student engagement. 


3. Research Questions 


This study helps fill this gap, anchoring its analytics and explanation in terms of 
evidenced-based procedures administered to undergraduate and graduate students across a 
series of university-level hybrid and online-only classes. This design allows us to examine 
two fundamental research questions. First, how does the peer review process influence 
students’ views in terms of the task, procedural, and process aspects of student engagement? 
Data collected from students prior to and then post-participation in the peer review process 
supports this analysis. Second, we assess how students’ perceptions of the peer review 
process vary depending on class format or duration. Data from the different course 
formats-e.g., online and hybrid support this analysis. Combined, these lines of study clarify 
theoretical aspects and applied implication of the peer review process with respect to 
promoting student engagement. 


4, Method 


4.1 Sample 

We designed a quasi-longitudinal, cross-sectional sample based on a series of courses that ran 
from Spring 2014 through Fall 2014. Over that span, the two authors delivered 8 courses-4 
hybrid and 4 completely online courses at a public University in the Mid-Atlantic region. The 
hybrid courses involved undergraduates, the latter MBAs. The first author taught all hybrid as 
well as 2 online sections; each of these classes was a survey course of the global business 
environment. This course is a senior-level requirement for all undergraduate international 
business majors, an elective for various majors in the University, and, with regards to the 
graduate population, an elective of the MBA program. The second author taught the remaining 
2 online sections. Both were the capstone Corporate Strategy course, a core requirement of 
the MBA program. Collectively these courses generated our sample of 272 students, 156 of 
whom took a hybrid course, and 116 of whom took the online-only course. 

4.2 Peer Assessment Procedures 


Logistically, all courses applied the peer assessment function provided by the Canvas LMS 
(Note 1). We configured this function, on a course by course basis, to assign peer review 
responsibilities to each student automatically. More precisely, when a student submitted his or 
her assignment, Canvas randomly generated a list of their assigned reviews and notified the 
student directly. Students then viewed their classmates’ submission via the e-reader, applied a 
preset, point-and-click rubric, and provided narrative feedback in the Comment Stream. We 
incorporated a preset rubric given reports that the presence of unambiguous evaluative 
criteria reduces subjectivity and increases inter-rater agreement (Falchikov & Goldfinch, 
2000). Regarding the rubric, it explicitly followed Bloom’s Taxonomy, asking the student to 
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evaluate each assigned submission in terms of its perfonnance with regards to its 
effectiveness in “Remembering, Understanding, Analyzing, Applying, Evaluating, and 
Creating” elements of the class content (Bloom, Engelhart, Furst, & Krathwohl, 1956). Each 
dimension applied a 5-point scale, running from Outstanding, Distinctive, Satisfactory, 
Deficient, and Missing. The score per dimension ran from a high of 4 (Outstanding) to a low 
of 0 (Missing). Regarding the Comment Stream, it was purposely open-ended with no preset 
stipulation of minimal expectations; students were charged with the general task of providing 
useful feedback to their classmates. 

All sampled courses used the Canvas LMS and, hence, applied the same peer assessment 
procedure. As a rule, students reviewed between 3 and 5 other classmates’ submissions on 
two different assignments: (1) an interim discursive assignment (a conventional specified 
paper on an aspect of the class content) and (2) a summary creative assignment (a so-called 
virtual field trip that required the student to develop a series of 7 image-slides along with a 50 
word or less narrative explanation per slide that represented the most meaningful ideas they 
had studied during the course). Each assignment had format regulations regarding its scope 
and scale. All students in the sampled sections reviewed 3 to 5 submissions of each type of 
assignment. 

We did not formally lecture on the principle of nor instruct the students on the procedure of 
conducting a peer review. Rather, the students were advised to review online resources, 
supplied by the instructor through the Canvas LMS, which explain the purpose of the task, 
elaborate its usefulness, and provide general recommendations on how to complete the 
assignment. Two reasons motivated this choice. First, online courses differ from conventional, 
face-to-face counterparts, lacking the platform to engage students personally when profiling 
an assignment. That is, in a conventional class, one can directly coach students on how to 
engage the intricacies of an unusual assignment, such as the peer review process. We lacked 
the same opportunity with respect to the online population. Moreover, both instructors 
designed their online class with an eye toward encouraging students to self-regulate their 
learning, incorporating a dynamic lesson plan that transferred considerable course 
responsibilities to the student. Including a module on how to conduct a peer review ran 
counter to that orientation. With regards to the hybrid formatted courses, maintaining 
consistency between that subsample as well as the online subsample led to applying the same 
procedure. 

4.3 Data 

At the start of each course, all students completed an online survey (hereafter, pretest) 
regarding their initial expectations of their pending experience with the peer assessment 
procedure. Then, at the end of the course, they completed an online survey (hereafter, posttest) 
that asked them about their experience. Items on both the pretest and posttest were derived 
from relevant literatures and in consultation with representatives of the host University’s 
Center for Teaching and Assessment of Learning. Some questions overlap between the 
pre-and post-surveys; others, given temporality issues, did not. Survey questions used 5-point 
Likert-scale, bounded by Strongly Agree and Strongly Disagree. Completing these surveys, 
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along with one other on aspects of class design and delivery, earned formal credit (a total of 1 
percent of the student’s final grade). In addition, we conducted follow-up interviews and 
discussions with students, both formally and informally, throughout the study. Relatedly, 
College directed course evaluations generated student commentary, some of which spoke of 
their judgment of the effectiveness of the peer assessment process. 

4.4 Analysis 

Initial data analyses evaluate the frequency distribution of the responses to the various survey 
measures. We explore the data beyond the full sample by segmenting the views of online and 
hybrid subsets. Survey research routinely uses Likert-style questionnaires. Still, debate 
persists regarding the use of parametric versus nonparametric tools. Review of our survey 
data did not reveal a systematic pattern of skew, either left or right-hand, for the full set as 
well as the subsets. Therefore, we use parametric tools, namely the t-test, given its power 
advantage over nonparametric alternatives when assessing a normal distribution. The 
pairwise t-test assessed differences in students’ views on the pre-test and the post-test, 
whereas the independent sample t-test was used to examine differences in course format 
(online vs hybrid). 

We regard Time and Place as categorical variables. Time was set by whether the course 
followed a traditional 14-week semester format versus a 7-week accelerated timeframe. Four 
of the classes were the fonner, 4 were accelerated. With respect to Place, the hybrid sections 
conducted 18 in-class, face-to-face sessions of 75 minutes each. In addition, the hybrid 
sections used an extensive series of online assignments, activities, and assessments staggered 
over the course of the 14 week semester. These online activities replaced 8 face-to-face class 
sessions. The 4 classes taught completely online relied exclusively on electronic contact 
between students and their classmates as well as with the professor. 

5. Results 

Response rates for the pretest and posttest surveys averaged 99% across the sampled sections 
(high of 100%, low of 97%). Table 1 reports the results for the research items with regards to 
student’s views on the pretest and posttest. In terms of the pretest, in anticipation of the peer 
review process, students consistently rated highly nearly all measures. The highest-rated 
measure on the pretest was a bit of a surprise—nearly 90% of the sample agreed that “In 
developing my understanding of the class materials, I expect that completing peer reviews of 
my classmates’ work will be helpful.” A corollary measure received strong support: 
“completing peer reviews will help me to understand what qualifies as outstanding versus 
poor performance” found agreement from 75% of the sample. Students regarded various 
indicators of the potential value of the peer review process positively. Nearly 70% of the 
sample expected its benefits to outweigh its demands and 80% saw the peer review processes 
providing a useful platform to develop the sorts of competencies sought in the workplace, 
notably providing and receiving criticism. Most students expected their classmates to do a 
good job providing them critical and useful feedback. Lastly, just 30 percent of the sample 
worried that completing their assigned peer reviews would take a lot of time and effort. 
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Table 1. Students’ view of the peer evaluation process. Pre-and post-test frequencies and tests 
of mean differences (N = 272) 


Item 

Source 

Percent Per Category 



Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Mean 

(5=High, 

l=Low) 

/SD 

Paired 

t-test: 

t-value/ 

p-value 

Overall, I expect that 
the benefits of the 
peer review process 
will outweigh its 
demands. 

Pre-Test 

0 

7.0 

23.6 

49.4 

19.9 

3.82/.83 

t= 3.364 

p = .001 

Post-Test 

3.2 

10.3 

27.0 

43.7 

15.9 

3.59/.98 

I’m worried that 
completing my 

assigned peer 

reviews will take a 
lot of time and 
effort. 

Pre-Test 

3.7 

33.2 

32.8 

24.7 

5.5 

2.95A97 

t = .323 
p = .747 

Post-Test 

6.3 

33.3 

29.4 

26.6 

4.4 

2.89/1.01 

Overall, I believe the 
process of peer 
reviewing will 

improve my ability 
to accept and give 
criticism to my 
classmates. 

Pre-Test 

0.4 

2.9 

10.9 

64.9 

20.9 

4.01/.69 

t = 4.74 

p - .000 

Post-Test 

3.2 

7.7 

19.4 

53.2 

16.7 

3.72A94 

I believe that 

formally reviewing 
my peer’s work will 
provide valuable 

experience and 

preparation for the 
workplace. 

Pre-Test 

0.4 

2.9 

18.5 

53.4 

24.8 

3.99A77 

t = 2.83 
p = .005 

Post-Test 

1.4 

9.0 

16.7 

52.7 

20.3 

3.81/.91 

I expect that my 
peers will do a good 
job providing me 
with critical and 
useful feedback. 

Pre-Test 

0.4 

5.4 

22.2 

57.7 

14.2 

3.80/.76 

t = 2.95 

P = .004 

Post-Test 

1.2 

7.5 

27.8 

53.6 

9.9 

3.64/.81 

I believe my 

classmates will 

objectively and 

fairly review my 
work. 

Pre-Test 

0 

2.9 

19.2 

62.8 

15.1 

3.90/.67 

t = .609 
p = .543 

Post-Test 

1.4 

4.5 

18.1 

57.5 

18.6 

3.87/.81 

I expect that 

completing peer 

reviews will help me 
to understand what 
qualifies as 

outstanding versus 
poor performance. 

Pre-Test 

2.4 

6.6 

18.6 

49.1 

23.4 

3.84A94 

t = -1.39 

p - .166 

Post-Test 

0.5 

5.0 

13.5 

50.9 

30.2 

4.05/.82 

In developing my 
understanding of the 
class materials, I 
expect that 

completing peer 

reviews of my 
classmates’ work 

will be helpful 

Pre-Test 

0 

4.2 

6.0 

58.1 

31.7 

4.17/.72 

t = 4.22 

p = .000 

Post-Test 

2.7 

13.5 

8.6 

46.4 

28.8 

3.85/1.07 


7 


www.macrothink.org/jei 





























,11 Macrothink 
All Institute" 


Journal of Educational Issues 

ISSN 2377-2263 
2015, Vol. 1, No. 1 


The results of the posttest indicate that for nearly all measures, the average student’s views 
changed, changed for the worse and, in several cases, changed significantly. Notably, after 
experiencing the peer review process, fewer students saw its benefits outweighing its 
demands. This indicator’s mean rating decreased 6%, falling from 3.82 on the pretest to 3.59 
on the posttest; this decrease was highly significant (p > .001). Mean values for items that 
measured students’ ability to adopt an executive perspective-namely, giving and accepting 
criticism that is part of parcel of the professional workplace-also declined significantly 
(p >.000) between the pretest to the posttest. Correspondingly, students’ expectation of 
receiving fair and objective criticism from their classmates declined after experiencing the 
peer assessment process. Table 1 shows that for only one measure did students’ agreement 
increase from the pretest to the posttest. Namely, “I expect that completing peer reviews will 
help me to understand what qualifies as outstanding versus poor performance” found greater 
support after students had participated in assessing their peers. In the pretest, 72.5 percent of 
the sample agreed; in the posttest, 81.1 percent did. This increase was not statistically 
significant. 

Assessing the data in terms of our control variables, time and place, followed the same 
analytic routine (See Tables 2a - Pre-test, and 2b - Post-test). Frequency analyses indicate 
negligible differences in the scale and scope of students’ views when controlling for time and 
place. Independent sample T-tests of the mean differences between the online/7-week 
sessions and the hybrid/full semester sessions confirmed this general tendency. Contrary to 
expectation, students’ perceptions of the peer review process did not differ significantly 
between online only versus hybrid formatted courses. However, the hybrid/full semester 
students indicated that the peer review process helped motivate them to take responsibility for 
their own learning to a significantly greater extent than did students in the accelerated online 
courses. 
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Table 2a. Online vs. hybrid frequencies and tests of mean differences. Students’ pre-test view 
of the peer evaluation process (N = 272) 


Item 

Source 

Percent Per Category 



Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Mean 

(5=High, 

l=Low) 

/SD 

Paired 

t-test: 

t-value/p- 

value 

Overall, I expect that 
the benefits of the 
peer review process 
will outweigh its 
demands. 

Online 

0 

7.8 

22.6 

47.8 

21.7 

3.85A86 

t = .20 

p = .840 

Hybrid 

0 

6.4 

24.4 

50.6 

18.6 

3.81/.81 

I’m worried that 
completing my 

assigned peer reviews 
will take a lot of time 
and effort. 

Online 

4.3 

29.6 

31.3 

28.7 

6.1 

3.03/1.00 

t = 1.07 

p = .284 

Hybrid 

3.2 

35.9 

34.0 

21.8 

5.1 

2.90/.95 

Overall, I believe the 
process of peer 

reviewing will 

improve my ability to 
accept and give 

criticism to my 

classmates. 

Online 

0 

7.2 

15.7 

54.2 

22.9 

3.93A82 

t = 1.52 

p = .131 

Hybrid 

0.6 

0.6 

8.3 

70.5 

19.9 

4.08/.60 

I believe that formally 
reviewing my peer’s 
work will provide 
valuable experience 
and preparation for 
the workplace. 

Online 

1.2 

6.1 

25.6 

39.0 

28.0 

3.87A94 

t = 1.65 

p = .101 

Hybrid 

0 

1.3 

14.7 

60.9 

23.1 

4.06/.65 

I expect that my peers 
will do a good job 
providing me with 
critical and useful 
feedback. 

Online 

0 

7.2 

25.3 

56.6 

10.8 

3.71/.76 

t = 1.31 

p = .192 

Hybrid 

0.6 

4.5 

20.5 

58.3 

16.0 

3.85A76 

I believe my 

classmates will 

objectively and fairly 
review my work. 

Online 

0 

3.6 

26.5 

55.4 

14.5 

3.81/.72 

t = 1.50 

p = .137 

Hybrid 

0 

2.6 

15.4 

66.7 

15.4 

3.95/.64 

I expect that 

completing peer 

reviews will help me 
to understand what 
qualifies as 

outstanding versus 

poor performance. 

Online 

2.4 

10.8 

20.5 

48.2 

18.1 

3.69A97 

t = -2.18 

p = .031 

Hybrid 

2.4 

2.4 

16.7 

50.0 

28.6 

4.00/.88 

In developing my 
understanding of the 
class materials, I 
expect that completing 
peer reviews of my 
classmates’ work will 
be helpful 

Online 

0 

7.2 

8.4 

44.6 

39.8 

4.17/.88 

t = .62 

p = .536 

Hybrid 

0 

1.2 

3.6 

71.4 

23.8 

4.18/.54 
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Table 2b. Online vs. Hybrid Frequencies and Tests of Mean Differences. Students’ post-test 
view of the peer evaluation process (N = 272) 




Percent Per Category 



Item 

Source 

Strongly 

Disagree 

Disagree 

Neutral 

Agree 

Strongly 

Agree 

Mean 

(5=High, 

l=Low) 

/SD 

Paired 

t-test: 

t-value/p 

-value 

Overall, I expect that the 
benefits of the peer 
review process will 
outweigh its demands. 

Online 

3.7 

12.1 

26.2 

37.4 

20.6 

3.59/1.06 

t= .01 

p = .991 

Hybrid 

2.8 

9.1 

27.3 

48.3 

12.6 

3.59A92 

I’m worried that 

completing my assigned 
peer reviews will take a 
lot of time and effort. 

Online 

7.5 

33.6 

28.0 

28.0 

2.8 

2.85/1.00 

t = -.56 

Hybrid 

5.6 

33.6 

29.4 

25.9 

5.6 

2.92/1.02 

p = .576 

Overall, I believe the 
process of peer reviewing 
will improve my ability 
to accept and give 
criticism to my 

classmates. 

Online 

2.6 

13.0 

23.4 

44.2 

16.9 

3.60/1.00 

t = .1.41 

Hybrid 

3.5 

4.9 

17.5 

57.3 

16.8 

3.79/.90 

p = .162 

I believe that formally 
reviewing my peer’s 
work will provide 

valuable experience and 
preparation for the 

workplace. 

Online 

1.3 

10.4 

19.5 

42.9 

26.0 

3.82A98 

T3 <-*- 

ii 

8 s 

OO 

Hybrid 

1.4 

8.4 

14.7 

58.7 

16.8 

3.81/.86 

I expect that my peers 
will do a good job 
providing me with 

Online 

1.9 

9.3 

26.2 

47.7 

15.0 

3.64/.91 

t = .14 

p - .885 

critical and useful 

feedback. 

Hybrid 

0.7 

6.3 

28.7 

58.0 

6.3 

3.63A73 

I believe my classmates 
will objectively and 

Online 

1.3 

5.2 

22.1 

53.2 

18.2 

3.81/.72 

t = -.72 

fairly review my work. 

Hybrid 

1.4 

4.2 

16.2 

59.2 

19.0 

3.95/.64 

p = .471 

I expect that completing 
peer reviews will help 
me to understand what 
qualifies as outstanding 
versus poor performance. 

Online 

1.3 

5.2 

15.6 

48.1 

29.9 

3.69A97 

t = -.66 

Hybrid 

0 

4.9 

12.6 

52.4 

30.1 

4.00/.88 

p = .510 

In developing my 

understanding of the 
class materials, I expect 
that completing peer 
reviews of my 

classmates’ work will be 
helpful 

Online 

2.6 

13.0 

11.7 

36.4 

36.4 

4.17/.87 

t = .59 

Hybrid 

2.8 

14.0 

7.0 

51.0 

25.2 

4.18/.54 

p = .550 


6. Discussion 

Mapping the elements and effectiveness of peer assessment attracts increasing attention. 
Consistently, studies and surveys highlight the role of students’ outlooks, cognitive and 
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affective motivations, and expectations as moderators of their engagement with the course. 
However, evidence-based research on students’ views of the potential and performance of 
peer review lags, thereby hindering our ability to translate the growing volume of anecdotes 
about students’ view of reviewing classmates into systematic knowledge that guides effective 
course design and delivery. Deficiencies are particularly glaring, given the fast-developing 
capabilities of leading LMSs, with respect to the context of hybrid and online education. 


Our results help resolve some of these concerns, especially with respect to the issue of 
student engagement. Again, the increasingly impersonal teacher-student relationship found in 
online and hybrid courses challenges conventional notions of engaging a student in a 
productive and meaningful learning experience. Given online content’s lack of the persuasive 
merits of professorial charisma, motivating students to make a “psychological investment in 
learning” is a daunting proposition. In online classes, efforts to mimic the social processes 
that mark the face-to-face classroom, via cyber-based relationship-building strategies, 
struggle (Nicol et. ah, 2014). Furthermore, geo-dispersion fosters benign anonymity, both 
among students as well as with the instructor (Santana, 2014). Similarly, asynchronous 
workflow makes it difficult to promote contact via real-time collaboration. The 
fundamentally different outlooks at play in a brick and mortar class versus an online course 
complicate both. Whereas face-to-face interactions anchor the value of classroom settings, 
online instructors must look to promote student engagement with a class design structure that 
precisely specifies activities, assignments, and assessments without unduly sacrificing 
flexibility, organization, and clear expectations (Crews & Butterfield, 2014; Dixson, 2012). 
This literature suggests that encouraging students in an online environment to engage the 
class, as well as each other, fundamentally depends on the design and delivery of the course 
content. Done well, students engage and make a psychological investment in learning. Done 
poorly, students disengage, drop-off, and, in extreme cases, dropout. 


In principle, our data indicate that the sampled students agreed that the peer review process 
provided a robust method to improve engagement. That is, students consistently recognized 
the potential value of the peer review process to making the class a more meaningful 
experience. Across the sampled sections, which spanned graduate and undergraduates, 
traditional versus accelerated semesters, as well as differentiated course content, students 
consistently affirmed the value of the opportunity to participate in peer assessment. On this 
theme, Eccles and Wigfield (1995) define task value as “the value a task acquires because it 
is instrumental in reaching a variety of long and short-range goals.” Corresponding measures 
in this study indicate that the sampled students recognize the intrinsic task value of the peer 
review process; measures asking about professional benefits, workflow requirements, and 
academic insights consistently score highly on the pre-and the posttest. With that in mind, if 
we accept the proposition that these outlooks are markers for student engagement, then it 
seems quite plausible that the peer review opportunity positioned students to benefit from the 
motivational, reflective, self-efficacy, autonomy, and self-regulated attributes of engagement 
(Boud & Falchikov, 2007; Richardson & Newby, 2010). 


11 


www.macrothink.org/jei 



M Macrothink 
All Institute™ 


Journal of Educational Issues 

ISSN 2377-2263 
2015, Vol. 1, No. 1 


6.1 Mismatch: Expectations vs. Experience 

Given the proceeding result, how then might we interpret the drop in students’ perception of 
the value of the peer review process-again, recall the slight but systematic decline in mean 
values on virtually all measures between the pretest and posttest. First and foremost, this 
result has broad precedent. A similarly designed investigation by Mulder et al. (2014) reports 
that “prior to engaging in peer review, students had remarkably high expectation of both the 
process and the competence of their peers as reviewers...However, a modest overall 
downward shift and positive perception was observed following peer review.” In many ways, 
our results echo the reported mismatch between expectations and outcomes. Mulder et al. 
(2014) struggling to pinpoint an explanation, recommend further research to isolate these 
issues and untangle the relationship. 

Our results, along with an ongoing series of conversations with students and colleagues, point 
to a likely explanation. Operationally, recall that we relied upon the Canvas LMS to organize 
and administer the peer evaluation process. Overall, the functionality of this procedure is 
markedly robust, supporting efficient administration for the instructor as well as 
straightforward engagement for the student. However, the Canvas procedure suffers a key 
limitation - one does not have the option to make the peer review process anonymous. That 
is, all peer reviewers know the identity of the students whose work they are reviewing. 
Likewise, all students know who has reviewed their work. 

The lack of anonymity enacts a potentially problematic situation. Research reports that 
interpersonal variables, in the absence of anonymity, likely moderate how students apply the 
peer review procedure (Van Gennip, Segers, & Tillema, 2009; Stepanyan et al., 2009). More 
precisely, factors such as peer pressure and fear of disapproval when assigning low scores can 
affect the outcomes of classmates’ respective assessments. Others point to various social 
dynamics as likely influencing students’ objectivity in assessing the quality of their 
classmates’ work (Panadero et al., 2013; Boud, 2000). Effectively, the implicit pressure to 
praise mediocrity, or in the least, avoid criticizing it, may, in turn, result in peer assessments 
that led some students to question the value, if not the legitimacy, of the peer review process. 
So, for example, the highly significant drop, from the pretest and posttest, in the mean value 
of items such as “In developing my understanding of the class materials, I expect that 
completing peer reviews of my classmates’ work will be helpful” or “Overall, I believe the 
process of peer reviewing will improve my ability to accept and give criticism to my 
classmates,” arguably speaks to some students’ sensitivity to this situation. The authors are 
conducting further research to unravel this relationship, questioning students about the 
importance of anonymity as well as developing measures to gauge the objectivity of the 
commentary provided from a peer reviewer to a classmate. 

6.2 Time and Place 

Interestingly, and quite honestly, surprisingly, neither time nor place influenced students’ 
views of the peer assessment process. In theory, time and place moderate patterns of student 
engagement, a key precursor of effective peer assessment (Carr, 2014; Nicol et. al., 2014). 
Orthodox face-to-face classes involve classmates progressively developing social routines 
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and interpersonal relations that anchor effective peer engagement. Social processes set and 
sustain the intricate relationship between emotion and cognition in the processes commonly 
linked to learning (Pekrun, 2006; Artino, 2009). Crews and Butterfield (2014), for example, 
report that students saw the most positive impact of face-to-face learning in terms of this 
facilitation of direct interaction in the context of lectures, discussions, and in-class group 
activities. Enabling students to find common ground with ostensible strangers within a 
face-to-face class establishes the basis for the confident interactions that support a student’s 
capacity to assess his or her peers. 

Furthermore, the face-to-face class setting requires students attend lectures or participate in 
class-based activities, thereby benefiting from the interaction provided by the physical 
learning environment, the classroom milieu, and the ethos of the wider University. Schutz, 
Hong, Cross, and Osbon (2006), for example, linked students’ emotional experiences to 
person-environmental transactions that take place within those sorts of settings, particularly 
those that emerge in the scripted structure of a classroom. Relatedly, social impact theory 
suggests that the effects of interpersonal variables should prove stronger in face-to-face 
classroom settings than in an online course given that that social influences, notably peer 
pressure, increase when others are physically proximate (Latane, 1981). Pope (2005) and 
Stepanyan et al. (2009) document this effect, finding that students in face-to-face classroom 
settings typically feel uncomfortable and experience stress when publicly evaluating their 
peers. 

In contrast, person-environment transaction in an online class differs dramatically. The 
defining “anytime, anyplace” instrumentality of an online class lets students engage a course 
and classmates wherever, wherever, and often, in whatever format, they prefer. Relatedly, 
classes delivered on-campus, either over a quarter or semester, run from 10 to 16 weeks, and 
span discrete sessions of sustained socialization. In contrast, it is not unusual for an online 
class to run, from start to finish, just 4 to 7 weeks, during which students will never “meet,” 
conventionally defined, a classmate. In cases where paths cross, most, if not all, do so via the 
Internet. 

Despite the suggestion that time and place should matter, our results indicate otherwise. 
Searching for precedent for this finding finds little. Tangential topics cover a range of issues, 
such as lecture length, traditional versus nontraditional students, course organization, or 
instructor enthusiasm and expertise. We suspect larger trends are afoot. Consider that in 2010, 
33% of US college students enrolled in at least one online class. Presently, more than half of 
all students in higher education have taken at least one online course. By 2020, forecasts call 
for roughly 75% of all college students to do so (Allen & Seaman, 2014). The increasing 
exposure and, presumably, increasingly sophisticated familiarity with online education 
platforms progressively socializes students to accept, if not prefer, self-direction in learning 
(Brock, 2015). 

Indeed, just as online education has reset the boundaries between learners and teachers, by 
weakening the centralization of expertise and the distribution of subject-matter authority, so 
too has it reset the boundaries between students and their class, as the design logic of 
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“anytime, anyplace” engagement fortifies their direct control of the course (Ponti, 2014; 
Brock, 2015). Essentially, students are increasingly empowered self-directed learners who, 
when given the opportunity to self-regulate their engagement of an LMS-centric course, no 
longer differentiate that engagement in terms of the duration or format of the course. 
Furthermore, the intrinsic design and delivery structure of an online course demands that 
students take “responsibility for directing their own learning, time management, keeping up 
with the class, completing the work on time and being active contributors to instruction” 
(Hung, Chou, Chen, & Own, 2010). Fundamentally, shifting from a teacher- to 
student-centric pedagogy requires higher conscientiousness and sustained self-discipline from 
a student in self-regulating motivation and goal achievement (Maehr, 1989; Brock, 2015). 
Moreover, all courses sampled in this study relied upon the Canvas FMS; this choice resulted 
in designing explicitly stipulated activities, assignments, and assessments in order to 
systematize the workflow. The imperatives of effective online course design, within the 
precise context of an FMS, inevitably expand the scale and scope of a student’s control of the 
learning process relative to that of the instructor (Candy, 1991; Brock, 2015). Essentially, our 
results suggest time and place no longer matter because the “anytime, anyplace” 
instrumentality of online education makes time and place no longer matter in the context of 
emergent modes of online education. 

7. Conclusion 

This research evaluates how students anticipate the opportunity and then interpret their 
experience of the potentially powerful pedagogy of peer assessment in an online environment. 
Our results, we believe, better infonn educational professionals on how to improve the design 
and delivery of assignments, activities, and assessments in ways that spur and sustain student 
engagement in an online environment. Fundamentally, we hold that it is the mission of all 
programs and instructors to design and deliver a range of activities, assignments, and 
assessments that promote student engagement. Compelling in any context, it is especially so 
in the online environment. The orthodox dyadic relationship between student and teacher, 
performed within the scripted environment of a face-to-face, bricks and mortar classroom, 
provides multiple methods to motivate student engagement. Online, these methods are lost, 
replaced by the benign impersonality of electronic milieus that, if not challenged, will 
confound and quite possibly corrupt students’ incentive to “learn to learn” (Corno & 
Mandinach, 1983). Preempting this threat calls for designing and delivering online courses 
explicitly geared toward promoting student engagement. 

With that in mind, the general theme of our results supports the value of peer review 
processes and highlights a means to improve its effectiveness. Regarding the former, the vast 
majority of our sample recognized and then confirmed the value of peer assessment across a 
range of markers of student engagement. In our sample, most students acknowledged the 
positive potential and useful performance of the peer review process in helping to understand 
what qualifies as outstanding versus poor performance. Absent concerted effort, students 
rarely see real-time examples of excellence as well as failure. The peer review process, by 
cracking open the black box of examples and standards, provided an effective means to 
improve the transparency of performance standards and evaluation. 
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Although students’ view of the value of peer evaluations declined between the pretest and the 
posttest, it is important to note the strength of student agreement in absolute as well as 
relative terms. Put differently, on average, for the individual student who questioned the value 
of the process, three others confirmed it. Looking forward, improving the usefulness of the 
peer evaluation to the ‘3,’ as well as demonstrating it to the skeptic, has a straightforward 
solution. Conversations with students indicate that making assessments anonymous would 
inspire greater objectivity and, presumably, greater value. To that end, growing research 
suggests that anonymity might offer a solution for the possible negative impact of 
problematic interpersonal variables on the perceived utility of peer assessment. Ainsworth et 
al. (2011) and Howard, Barrett, and Frick (2010) found that anonymity, in terms of giving 
feedback in the context of an asynchronous web forum, made students approximately five 
times more likely to provide substantively critical feedback than those whose identities were 
known to the recipients. Similarly, Vanderhoven, Raes, Montrieux, Rotsaert, and Schellens 
(2015) “found that pupils felt more positive towards peer assessment and experienced less 
peer pressure and fear of disapproval when scores were given anonymously using CRT.” 

On a practical basis, student learning is influenced, positively as well as negatively, by the 
design and delivery of activities, assignments, and assessments. Our findings, we believe, 
provide useful guidance to instructors in their consideration of incorporating peer evaluation 
processes into their course. Operationally, our profile of the strengths and weaknesses of the 
peer review process delivered via the Canvas LMS highlights its remarkable administrative 
ease. But, our results also point to a particularly problematic limitation, namely the issue of 
non-anonymity. Developing a workaround, in the event that Canvas does not offer that option, 
would improve the effectiveness of the peer review process to set and sustain a higher order 
level of student engagement. Absent that feature, our results caution instructors to anticipate a 
likely mismatch between students’ view of the potential and then perfonnance of the peer 
review process. 

8. Future Research 

Improving our understanding calls for future research to model and assess factors and 
relationships that elaborate the demographic and self-reported data explored here. Potentially 
useful directions include developing teaching strategies or learning systems to manage the 
interpersonal behaviors that likely moderate non-anonymous peer evaluation process. 
Similarly, our interesting findings on the irrelevance of time and place endorse evaluating the 
fit of current course designs, both online as well as hybrid, with an eye toward the “anyplace, 
anytime” dynamic of self-directed learning and its implication to student engagement. 
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Note 1. The Canvas LMS is a cloud-native software package provided by Instructure, an 
educational technology company. See http://www.canvaslms.com/higher-education for a 
fuller profile. 
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